GetModProps issues

It looks like my solution to the "ModuleProperties" problem was not in fact solved here:
[https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14907318&#14907318]

On large modules, that routine is taking 3.5 seconds per call and not the 13ms per call in that thread. Using v9.3.0.6.

I hate it when that happens.

-Louie
llandale - Mon Jan 07 14:21:30 EST 2013

Re: GetModProps issues
Mathias Mamsch - Thu Jan 10 07:55:40 EST 2013

Did you compare the three methods of the thread (i.e. using eval_ vs. normal getProperties) and how do the 3.5 seconds relate to the time, that the module needs for opening. I think that time might still be reasonable. Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Re: GetModProps issues
llandale - Thu Jan 10 10:33:54 EST 2013

Mathias Mamsch - Thu Jan 10 07:55:40 EST 2013
Did you compare the three methods of the thread (i.e. using eval_ vs. normal getProperties) and how do the 3.5 seconds relate to the time, that the module needs for opening. I think that time might still be reasonable. Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

OK,
  • small modules are still very fast (15ms)
  • medium ones slow(3500ms)
  • huges ones hopeless(13,000ms).
Whether the module is baselined and surprisingly whether the module is open seems to have no effect. These times are faster than opening the module, but it you query 3 attrs of the same module it is slower. The whole idea is to significantly improve speed over opening the module.

I'm guessing these times correlate to the time it takes to do a GUI manual "Module Properties", for our Huge modules takes several minutes.

-Louie

Re: GetModProps issues
Mathias Mamsch - Fri Jan 11 04:56:43 EST 2013

llandale - Thu Jan 10 10:33:54 EST 2013
OK,

  • small modules are still very fast (15ms)
  • medium ones slow(3500ms)
  • huges ones hopeless(13,000ms).
Whether the module is baselined and surprisingly whether the module is open seems to have no effect. These times are faster than opening the module, but it you query 3 attrs of the same module it is slower. The whole idea is to significantly improve speed over opening the module.

I'm guessing these times correlate to the time it takes to do a GUI manual "Module Properties", for our Huge modules takes several minutes.

-Louie

Well from one ModuleProperties you can read all three attributes. You will need to introduce a cache in your function, so the module properties will only read once per module. Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Re: GetModProps issues
llandale - Fri Jan 11 10:45:02 EST 2013

Mathias Mamsch - Fri Jan 11 04:56:43 EST 2013
Well from one ModuleProperties you can read all three attributes. You will need to introduce a cache in your function, so the module properties will only read once per module. Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

I'm thinking now that the eval_ can just get a Skip list of all module-level attribute values (except Attr-DXL) from the module. Since getting the ModuleProperties is where the time is spent; retrieving all values takes no additional time.

Re: GetModProps issues
llandale - Sat Jan 12 14:36:29 EST 2013

I re-wrote it. Constructive criticism is encouraged, including in the comments:

Date    fDateOf(string in_sDate)
{     Date    dat = in_sDate
        return(dat)
}
 
/*      fModProps_      Get Module-level attribute values without opening the module
        This uses the new DOORS v9 notion of "ModuleProperties".
        However, getting the ModuleProperties of large modules inexplicably takes almost as much
                time as it takes to open the module in the first place.  Getting the attr value
                of more than one such attr makes the whole notion useless.
                The culprit is getting the ModuleProperties handle using getProperties().
        This set of functions gets all the module-level attributes for a module when any
                is requested, and any subsequent retrieval by-passes the ModuleProperties and
                takes negligible time.
        If the user requests the value of some other module, this module's values are no longer remembered.
        Thus, time is saved presuming that the calling script gets all the relevant values from a single module
                before moving on to the next module.
*/
string  g_fModProps_NameModule  = ""          // FullName of module associated with the following Skip
Skip    g_fModProps_skpValues   = null          // module-level attr values KEY: 'string' NameAttr; DATA: 'string' value
 
//********************************************************
Skip    fModProps_GetAllValues(ModName_ in_mn)
{
        Skip    skpReturnValues = createString()
        string  ErrMess, EvalResult
 
        string  EvalCode = "
                ModName_        mn = addr_ " (int (addr_ in_mn)) "
                ModuleVersion   mv = moduleVersion (mn)
                ModuleProperties mp
                AttrDef         ad
                string          NameAttr, AttrValue
                Skip            skpValues       = addr_ " (int (addr_ skpReturnValues)) "
int TimeStart = getTickCount_
                getProperties (mv, mp) 
print \"\t>>fModProps_GetAllValues, getProperties.. \" (getTickCount_ - TimeStart) \"ms\n\"
                delete(mv)
                if (!null mp)
                {
                   for NameAttr in mp do
                   {  ad = find(mp, NameAttr)
                      if (null ad)                              continue
                      if (ad.dxl)                               continue        // Attr-DXL
                      noError()
                      AttrValue = mp.NameAttr \"\"
                      lastError()
                      put(skpValues, NameAttr, AttrValue)
                   }  // end module-level attrs
                }
                return_(\"\")
        "  // end string EvalCode
        ErrMess         = checkDXL(EvalCode)
        if (!null ErrMess)
        {  print "fModProps_GetAllValues(); checkDXL() error [" ErrMess "\t]... for code:" EvalCode "\n"
        }
        else
        {  noError()
int TimeStart = getTickCount_
           EvalResult   = eval_(EvalCode)
           ErrMess      = lastError() 
print "\t>>fModProps_GetAllValues, eval_ .. " (getTickCount_ - TimeStart) "ms\n"
           if (!null ErrMess)
           {  print "fModProps_GetAllValues() eval_() error [" ErrMess "\t]... for code:" EvalCode "\n"
              EvalResult        = ""
           }
        }
        // print "EvalResult [" EvalResult "]\n"
        return skpReturnValues
}    // end fModProps_GetAllValues()
 
//********************************************************
string  fModProps_GetValue(string in_NameModFull, string in_NameAttr) 
{    // Get the module-level attribute value of the module without opening it.
        // Attr-DXL values are not retrieved.
        // Return null on any error such as no-such module-level attr in that module
        if (null in_NameModFull or null in_NameAttr)    then return("")
        ModName_        mn = module(in_NameModFull)
        string  AttrValue
        if (null mn)                                    then return("")
int TimeStart = getTickCount_
        if (g_fModProps_NameModule != in_NameModFull)
        {            // First Request for this module.
                        // Retrieve ALL module-level attr values for this module.
           if (!null g_fModProps_skpValues) then delete(g_fModProps_skpValues)
           g_fModProps_skpValues        = fModProps_GetAllValues(mn)
           g_fModProps_NameModule       = in_NameModFull
        }
        else{}                // We already know all module-level attr values for this module
print "\t>>fModProps_GetValue, \t" (getTickCount_ - TimeStart) "ms\t" in_NameAttr "\n"
        if (find(g_fModProps_skpValues, in_NameAttr, AttrValue)) //
        then return(AttrValue)
        else return("")
}    // end fModProps_GetValue()
 
 
//-------------------------------------------------------------------
Date    fModProps_GetValue(string in_NameModFull, string in_NameAttr)
{    // Overloaded, presume the value represents a Date
 
        return(fDateOf(fModProps_GetValue(in_NameModFull, in_NameAttr)))
}    // end Date fModProps_GetValue()
 
 
 
 
void    DoTest(string in_NameModFull)
{
        Date    LastModOn = fModProps_GetValue(in_NameModFull, "Last Modified On")
        string  LastModBy = fModProps_GetValue(in_NameModFull, "Last Modified By")
        print in_NameModFull "\t" LastModOn " \t" LastModBy "\n"
}
 
                // Select a couple modules in the DOORS Explorer:
Skip    skpSelected = getSelectedItems()
Item    itm
for itm in skpSelected do
{  DoTest(fullName(itm))
}
delete(skpSelected)


-Louie

Re: GetModProps issues
Mathias Mamsch - Sat Jan 12 15:44:31 EST 2013

llandale - Sat Jan 12 14:36:29 EST 2013

I re-wrote it. Constructive criticism is encouraged, including in the comments:

Date    fDateOf(string in_sDate)
{     Date    dat = in_sDate
        return(dat)
}
 
/*      fModProps_      Get Module-level attribute values without opening the module
        This uses the new DOORS v9 notion of "ModuleProperties".
        However, getting the ModuleProperties of large modules inexplicably takes almost as much
                time as it takes to open the module in the first place.  Getting the attr value
                of more than one such attr makes the whole notion useless.
                The culprit is getting the ModuleProperties handle using getProperties().
        This set of functions gets all the module-level attributes for a module when any
                is requested, and any subsequent retrieval by-passes the ModuleProperties and
                takes negligible time.
        If the user requests the value of some other module, this module's values are no longer remembered.
        Thus, time is saved presuming that the calling script gets all the relevant values from a single module
                before moving on to the next module.
*/
string  g_fModProps_NameModule  = ""          // FullName of module associated with the following Skip
Skip    g_fModProps_skpValues   = null          // module-level attr values KEY: 'string' NameAttr; DATA: 'string' value
 
//********************************************************
Skip    fModProps_GetAllValues(ModName_ in_mn)
{
        Skip    skpReturnValues = createString()
        string  ErrMess, EvalResult
 
        string  EvalCode = "
                ModName_        mn = addr_ " (int (addr_ in_mn)) "
                ModuleVersion   mv = moduleVersion (mn)
                ModuleProperties mp
                AttrDef         ad
                string          NameAttr, AttrValue
                Skip            skpValues       = addr_ " (int (addr_ skpReturnValues)) "
int TimeStart = getTickCount_
                getProperties (mv, mp) 
print \"\t>>fModProps_GetAllValues, getProperties.. \" (getTickCount_ - TimeStart) \"ms\n\"
                delete(mv)
                if (!null mp)
                {
                   for NameAttr in mp do
                   {  ad = find(mp, NameAttr)
                      if (null ad)                              continue
                      if (ad.dxl)                               continue        // Attr-DXL
                      noError()
                      AttrValue = mp.NameAttr \"\"
                      lastError()
                      put(skpValues, NameAttr, AttrValue)
                   }  // end module-level attrs
                }
                return_(\"\")
        "  // end string EvalCode
        ErrMess         = checkDXL(EvalCode)
        if (!null ErrMess)
        {  print "fModProps_GetAllValues(); checkDXL() error [" ErrMess "\t]... for code:" EvalCode "\n"
        }
        else
        {  noError()
int TimeStart = getTickCount_
           EvalResult   = eval_(EvalCode)
           ErrMess      = lastError() 
print "\t>>fModProps_GetAllValues, eval_ .. " (getTickCount_ - TimeStart) "ms\n"
           if (!null ErrMess)
           {  print "fModProps_GetAllValues() eval_() error [" ErrMess "\t]... for code:" EvalCode "\n"
              EvalResult        = ""
           }
        }
        // print "EvalResult [" EvalResult "]\n"
        return skpReturnValues
}    // end fModProps_GetAllValues()
 
//********************************************************
string  fModProps_GetValue(string in_NameModFull, string in_NameAttr) 
{    // Get the module-level attribute value of the module without opening it.
        // Attr-DXL values are not retrieved.
        // Return null on any error such as no-such module-level attr in that module
        if (null in_NameModFull or null in_NameAttr)    then return("")
        ModName_        mn = module(in_NameModFull)
        string  AttrValue
        if (null mn)                                    then return("")
int TimeStart = getTickCount_
        if (g_fModProps_NameModule != in_NameModFull)
        {            // First Request for this module.
                        // Retrieve ALL module-level attr values for this module.
           if (!null g_fModProps_skpValues) then delete(g_fModProps_skpValues)
           g_fModProps_skpValues        = fModProps_GetAllValues(mn)
           g_fModProps_NameModule       = in_NameModFull
        }
        else{}                // We already know all module-level attr values for this module
print "\t>>fModProps_GetValue, \t" (getTickCount_ - TimeStart) "ms\t" in_NameAttr "\n"
        if (find(g_fModProps_skpValues, in_NameAttr, AttrValue)) //
        then return(AttrValue)
        else return("")
}    // end fModProps_GetValue()
 
 
//-------------------------------------------------------------------
Date    fModProps_GetValue(string in_NameModFull, string in_NameAttr)
{    // Overloaded, presume the value represents a Date
 
        return(fDateOf(fModProps_GetValue(in_NameModFull, in_NameAttr)))
}    // end Date fModProps_GetValue()
 
 
 
 
void    DoTest(string in_NameModFull)
{
        Date    LastModOn = fModProps_GetValue(in_NameModFull, "Last Modified On")
        string  LastModBy = fModProps_GetValue(in_NameModFull, "Last Modified By")
        print in_NameModFull "\t" LastModOn " \t" LastModBy "\n"
}
 
                // Select a couple modules in the DOORS Explorer:
Skip    skpSelected = getSelectedItems()
Item    itm
for itm in skpSelected do
{  DoTest(fullName(itm))
}
delete(skpSelected)


-Louie

Two things:
  • It might make sense to construct the eval code in a buffer and move the static parts to global constants, instead of reconstructing it with each call.
  • It is very dangarous to put strings that were constructed in an eval context to a skip list and use them after the eval context ended. You need to be sure that each string is permanently persisted in the string table. I guess in your case this might be the case, but the code can easily break when you do changes later. You should use a Buffer to transfer string data out of an eval context, however that will make your function even more complicated. Another thing I did, is putting the strings to a buffer and use stringOf to receive them back, before I put them to the skip. This way I know the string will be persisted in the string table. Kind of a bad hack, but we are in bad hack land anyway here.

Regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Re: GetModProps issues
llandale - Mon Jan 14 10:47:42 EST 2013

Mathias Mamsch - Sat Jan 12 15:44:31 EST 2013
Two things:

  • It might make sense to construct the eval code in a buffer and move the static parts to global constants, instead of reconstructing it with each call.
  • It is very dangarous to put strings that were constructed in an eval context to a skip list and use them after the eval context ended. You need to be sure that each string is permanently persisted in the string table. I guess in your case this might be the case, but the code can easily break when you do changes later. You should use a Buffer to transfer string data out of an eval context, however that will make your function even more complicated. Another thing I did, is putting the strings to a buffer and use stringOf to receive them back, before I put them to the skip. This way I know the string will be persisted in the string table. Kind of a bad hack, but we are in bad hack land anyway here.

Regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Your point #2; did you say that if I send the eval_ a Buffer, and can guarantee string table use in the main context by doing this?:
  • Buffer bufAttrVal = addr_" (int (addr_ in_bufAttrVal)) "
  • ...
  • bufAttrVal = mp.NameAttr \"\"
  • put(skpValues, NameAttr, stringOf(bufAttrVal))
  • // I guess I also need bufAttrName ..

I don't see how that would work, but then I don't see how any of this "voodoo" really works.

On another topic, I see that my noError/lastError pair does NOT trap "no access to read attr" DXL errors. I thus had to add a "if (canReadVal(ad)).." check.

-Louie

Re: GetModProps issues
Mathias Mamsch - Tue Jan 15 16:52:34 EST 2013

llandale - Mon Jan 14 10:47:42 EST 2013
Your point #2; did you say that if I send the eval_ a Buffer, and can guarantee string table use in the main context by doing this?:

  • Buffer bufAttrVal = addr_" (int (addr_ in_bufAttrVal)) "
  • ...
  • bufAttrVal = mp.NameAttr \"\"
  • put(skpValues, NameAttr, stringOf(bufAttrVal))
  • // I guess I also need bufAttrName ..

I don't see how that would work, but then I don't see how any of this "voodoo" really works.

On another topic, I see that my noError/lastError pair does NOT trap "no access to read attr" DXL errors. I thus had to add a "if (canReadVal(ad)).." check.

-Louie

You do not need to send the eval_ a Buffer, it is sufficient to create one in the eval_ context and use it for "stringOf"-ing every string that you want to persist after the eval_ has ended. I have used this technique in the past. Regards, Mathias


Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS